Repository navigation
fix(services): start ml jobs with the corvus-job entry point - #78
Merged
Merged
Conversation
The job runners defaulted to the command "corvus-ml", which is the package name and not an executable, so every Kubernetes job failed with "executable file not found in $PATH". The ml image provides the console script corvus-job. Change both defaults, fix the reference docs and add tests that keep the defaults in sync with ml/pyproject.toml and the ml/Dockerfile ENTRYPOINT.
Job pods mounted nothing, so feature builds wrote into the container layer and the node's ephemeral storage. Mount an emptyDir at /scratch (size limit configurable, default 20Gi) and point CORVUS_JOB_SCRATCH_DIR and TMPDIR at it, as docs/reference/ml-jobs.md recommends.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Jobs started from the UI fail immediately in 0.3.0 with
exec: "corvus-ml": executable file not found in $PATH. The runners defaultedjob_k8s_command/job_local_commandto["corvus-ml"](the package name); thecorvus-mlimage's ENTRYPOINT and console script iscorvus-job.Changes
["corvus-job"](settings,docs/reference/control-plane-api.md).corvus-job; contract tests assert the default is a[project.scripts]entry ofml/pyproject.tomland equals theml/DockerfileENTRYPOINT (skipped ifml/is absent).emptyDirat/scratch(size limitCORVUS_API_JOB_K8S_SCRATCH_SIZE_LIMIT, default20Gi) withCORVUS_JOB_SCRATCH_DIRandTMPDIRpointing at it. Unit tests cover the manifest, and that the job type is the first arg and equalsCORVUS_JOB_TYPE.CORVUS_API_URL/token come from the env secret (as in the helm values docs). The root filesystem is not read-only and the image setsHOME=/tmp, so no further cache dirs were needed.Services tests, ruff, ruff format and mypy pass locally.